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Abstract: This paper describes an application of learning analytics that builds on an existing 
research program investigating how students contribute and attend to the messages of others 
in asynchronous online discussions. We first overview the E-Listening research program and 
then explain how this work was translated into analytics that students and instructors could 
use to reflect on their discussion participation. Two kinds of analytics were designed: some 
embedded in the learning environment to provide students with real-time information on their 
activity in-progress; and some extracted from the learning environment and presented to 
students in a separate digital space for reflection. In addition, we describe the design of an 
intervention though which use of the analytics can be introduced as an integral course activity. 

Findings from an initial implementation of the application indicated that the learning analytics 
intervention supported changes in students' discussion participation. Five issues for future 
work on learning analytics in online discussions are presented. One, unintentional versus 
purposeful change; two, differing changes prompted by the same analytic; three, importance 
of theoretical buy-in and calculation transparency for perceived analytic value; four, affective 
components of students' reactions; and five, support for students in the process of enacting 
analytics-driven changes. 
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1. INTRODUCTION 

The field of learning analytics is concerned with the collection and analysis of data traces related to 
learning in order to inform and improve learning processes and/or their outcomes (Siemens et al., 
2011). Within this space, a distinction can be made between classes of analytics based on the types of 
data collected and the kinds of decision making targeted (Ferguson, 2012). At a macro level, 
administrators and policy makers have the opportunity to use learning analytics to make programmatic 
or legislative decisions. In such situations, data on past learning events is used to make decisions about 
future ones; these choices tend to be based on relatively long data time-cycles (Clow, 2012), affect large 
numbers of people, and involve outcome-type data; for example, summative assessments, performance 
indicators, and the like (Buckingham Shum & Ferguson, 2012). For these purposes, data may also be 
aggregated at various levels (i.e., student, class, department, institution, et cetera). 

In contrast, at a micro level, learners and teachers have the opportunity to use learning analytics to 
make more local decisions about the current learning events in which they are involved (Clow, 2012). In 
this case, the relevant data relates to tracking learning processes and an important element of 
interpretation is having a model of learning for the particular environment — i.e., a research-based 
framework for understanding what productive activity in the specific learning context looks like. In some 
cases, the model may be specified such that analytics data is processed and interpreted according to 
some system of rules leading to automatic changes in the learning system (e.g., Roll, Aleven, & 
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Koedinger, 2010). In other cases, data may be processed into categories according to the model, but 
then presented to stakeholders (students, teachers, administrators, et cetera) to support decision¬ 
making (e.g., Jovanovic et al., 2008). 


This paper reports on a learning analytics application of the latter type designed to support decision¬ 
making by students and teachers involved in asynchronous online discussions as a learning activity. The 
work builds on an existing research program investigating how students contribute and attend to the 
messages of others in online discussions. Both contributing and attending to discussion messages are 
important activities in realizing the theoretical potential of online discussions to support knowledge 
construction (Wise, Speer, Marbouti, & Hsiao, 2013). However a substantial research base shows that in 
actual asynchronous discussions learners often pay limited attention to others' posts (Hewitt, 2003; 
Thomas, 2002), resulting in conversation patterns that can be characterized as incoherent rather than 
dialogic (Herring, 1999; Webb, Jones, Barker, & van Schaik, 2004). Early research into these problems 
reported disturbingly low overall statistics, suggesting that the problems were global and, to some 
extent, systemic products of asynchronous discussion environments (Hewitt, 2003; Kear, 2001; Swan, 
2003). These findings spurred efforts to improve online discussion tools to support productive 
engagement and discussion (e.g., Scardamalia, 2004; Marbouti, 2012). However, more recent work 
disaggregating data across individuals has revealed that students in fact engage in very different kinds of 
behaviours when they participate in asynchronous online discussions (Wise, Perera, Hsiao, Speer, & 
Marbouti, 2012; Wise, Speer et al., 2013; Wise, Hsiao, Marbouti, Speer, & Perera, 2012). This suggests 
that discussion participation can also be improved with more targeted efforts to provide guidance to 
students individually using learning analytics. 


The paper describes how we attempted to take up this opportunity by using our pre-existing research 
program to inform the development of a learning analytics application for online discussions. We begin 
by laying out our theoretical framework and empirical findings about "listening" (attending to others' 
posts) in online discussions. We then explain how this work was translated into guidelines for practice 
and analytic metrics with which students and instructors could assess and reflect on their discussion 
participation. We describe two classes of analytics for online discussions that we developed — 
embedded and extracted analytics — as well as the design of the larger intervention framing their use. 
The latter half of the paper then describes and presents findings from an initial implementation of the 
learning analytics application and its implications for the future design of analytics interventions for 
asynchronous online discussions. 

2. FROM RESEARCH TO ANALYTICS 


The increasing amount of information automatically captured by online learning environments is 
attractive as a source of data with which to better understand and support learning processes. However, 
there is a wide gap between the kinds of data easily capturable in learning environments such as online 
discussions and the kinds of constructs established as pedagogically valuable (Buckingham Shum & 
Ferguson, 2012). To address this problem, it is vital to build bridges from both directions: for learning 
analytics researchers to draw on existing educational research and theory as they develop their 
applications; and for educational researchers to leverage data mining and information visualization work 
to explore how they might develop useful learning analytics based on their research programs. Here we 
describe our experience of the latter process with the hope that it can inform other educational 
researchers moving into the "middle space" of learning analytics. 
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2.1 Theoretical Framework and Research Base 


In this work, online discussions are conceptualized from a social constructivist perspective as a venue in 
which learners can interact to build both collective and individual understanding through dialogue 
(Kamel Boulos & Wheeler, 2007; Stahl, 2005). Scholars differ in their theoretical accounts of the 
mechanisms underlying the learning process, referring variously to the importance of being exposed to 
multiple viewpoints, articulating one's own ideas, experiencing socio-cognitive conflict, and the 
negotiation and internalization of group understandings (Lipponen, 2002; Stahl, 2005; Wise, Speer et al., 
2013). However, at a fundamental level all explanations depend on two basic processes in which 
learners must engage: "speaking" (externalizing one's ideas by contributing posts to the discussion); and 
"listening" (taking in the externalizations of others by accessing existing posts) (Wise, Speer et al., 2013). 
Speaking in online discussions is clearly visible to others; this may explain why the bulk of research on 
and guidance for participation in online discussions is focused on posting activity (Hew, Cheung, & Ng, 
2010). However, while listening is largely invisible, it is also critical for discussions that build 
understanding in the ways described above (Wise, Speer et al., 2013). The E-Listening Project is a four- 
year research effort funded by the Social Sciences and Humanities Research Council of Canada to 
understand better how students attend to others' contributions in online discussion. Work on this 
project has documented many of the different kinds of listening behaviours in which students engage 
(Wise, Perera et al., 2012; Wise, Speer et al., 2013; Wise, Hsiao et al., 2012) and has shown empirical 
connections between students' listening and speaking behaviours (Wise, Hausknecht, & Zhao, 2013; 
Wise, Hausknecht, &Zhao, 2014). 

While the language of speaking and listening draws on a metaphor based in face-to-face conversations, 
online discussions offer different affordances and constraints for these activities (Wise, Hausknecht, & 
Zhao, 2014; Wise, Perera et al., 2012). Specifically, in asynchronous online discussions, learners have 
greater control over the timeline and pace of their engagement (Jonassen & Kwon 2001). This creates 
opportunities for thoughtful listening and reflective speaking (Lipponen, 2002), but also additional 
challenges of time management, especially for prolific discussions (Peters & Hewitt, 2010). For this 
reason, helping learners to actively monitor and regulate how they speak and listen in online discussions 
is an important tool for supporting productive engagement in discussions. 

Given the above-described goal of using dialogue to build individual and collective understandings and 
the existing research base on asynchronous online discussions, particular speaking and listening 
behaviours can be characterized as more or less productive. Here we summarize the different 
dimensions of listening and speaking identified by the E-Listening project, relating them to the online 
discussion literature more broadly. First, in terms of speaking quantity, multiple posts are needed to 
respond to others' ideas and questions, elaborate on the points made, and negotiate with others (Pena- 
Shaff & Nicholls, 2004). These posts should be distributed throughout the discussion (rather than 
concentrated at the start or end), relating to the temporal distribution of participation. In addition posts 
of moderate length best support rich dialogue since very short posts tend to be shallow in their 
presentation of ideas (Dennen, 2001) but long posts are often perceived as overwhelming and are thus 
not read (Peters & Hewitt, 2010). Precise specifications of moderate length may differ by context and 
age-level; in higher education, this is often specified at around 100 to 200 words. In terms of speaking 
quality, posts that are clear, critical, and connected to the existing conversation support the generation 
of insight and understanding of the topic (Rovai, 2007). Posts whose arguments are based on evidence 
and/or theory can also trigger others to build on or contest the points productively (Clark, Sampson, 
Weinberger, & Erkens, 2007), and responses that clarify points, elaborate or question existing ideas, or 
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synthesize different ideas together help deepen the exploration of ideas and move the discussion 
forward (Pena-Shaff & Nicholls, 2004). In terms of listening activity, there are also multiple dimensions 
requiring attention. Considering breadth of listening, viewing a greater proportion of others' posts 
exposes students to a greater diversity of ideas (Wise, Hsiao et al., 2012). Depth of listening is important 
as an indication of the degree to which learners consider the ideas of others (Wise, Hsiao et al., 2012); 
greater listening depth is predictive of richer argumentation in posts made (Wise, Hausknecht, & Zhao, 
2014). Listening reflectivity (revisiting one's own and others' posts from earlier in the discussion) can 
provide context for interpreting recent posts and examine how thinking has changed. Revisiting others' 
posts has been shown to be predictive of more substantive responses to others' ideas and possibly 
increased reflection on how one's own ideas have changed (Wise, Hausknecht, & Zhao, 2014). Finally 
temporal distribution of listening is important since engaging in multiple sessions during a discussion and 
integrating listening and posting in the same session can support learners in making connections 
between ideas and contributing posts that productively build on previous ones (Wise, Speer et al., 
2013). 


2.2 Issues in Translating a Research Program into Learning Analytics 

While established learning research programs have much to offer the development of learning analytics, 
the process of translating research methodologies and measures into useful learning analytics metrics is 
far from trivial. Research programs work to identify traces of learning that are meaningful to 
researchers; however, such measures are often quite complex and/or theoretically grounded. They thus 
may not be intuitively meaningful (or meaningful in the same way) to students, instructors, and 
administrators. For this reason educational researchers need to go through a process of selection (and 
possibly modification) of metrics to make them understandable and useful to the target audience. There 
may also be opportunities to support the use of more complex analytics through the intervention 
design: how the analytics are introduced into the learning environment and how the activity of 
interpreting and making decisions based on them is framed. This raises another important 
consideration: while research programs generally use measures to understand what has happened in a 
learning event with eventual consequences for practice, in an analytics context stakeholders use metrics 
to make immediate decisions about their behaviour. Thus, researchers must carefully consider the 
anticipated effects of providing certain metrics to learners and teachers at particular points in the 
learning process. Finally, in choosing measures, it is also important to consider the accelerated timeline 
of data collection, processing, and interpretation required for learning analytics compared to research 
analyses. 


Once measures have been decided on, additional work is required to figure out how to present the 
traces of learning to students and teachers in a form that will be useful to them and frame the process 
of interpretation and decision-making as an integrated aspect of the learning activity tied to goals and 
expectations. While not specific to the case of translating a learning research program into analytics, 
these aspects of analytics design are as important as the selection of the measures. In the following 
sections, we describe all three aspects of our learning analytics application: the traces of learners' 
activity in online discussions we chose to capture, how we presented these traces to learners, and how 
we framed the inclusion of analytics as part of discussion activity to guide their use in productive 
decision-making by learners and teachers. 
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2.3 Two Kinds of Approaches to Learning Analytics 

Below we describe two different classes of analytics for learners that we developed for online 
discussions: embedded analytics and extracted analytics. Embedded analytics are traces of activity 
integrated into the learning environment itself that can be used by learners in real-time to guide their 
participation. In this case, interpretation of the analytics and participation in the learning activity are 
unified as a single endeavor. An advantage to embedded analytics is that they can be used seamlessly to 
support metacognitive monitoring during participation in the learning activity itself. However, a 
weakness of being embedded is the possibility of being ignored; i.e., there is no reason to assume that 
students will naturally use the analytic affordances of the tool to support their participation individually 
or as a group. For this reason, such use needs to be specifically encouraged by structuring it in to the 
learning activity parameters. 

Extracted analytics are traces of activity extracted from learning environment and presented back to 
learners for interpretation as a separate exercise from participating in the learning activity itself. That is, 
while the presentation of the analytics may be integrated into the overall learning environment, the 
activity of interpreting them is separate from that of engaging in the learning activity. For example, 
analytics presented to learners via most dashboard systems fall into this second category. 1 The 
distinction between embedded and extracted analytics will be further clarified through the description 
of the specific instantiation of each that follows in the subsequent sections. 

2.3.1 Embedded Analytics 

In this work, we have chosen to use a specific asynchronous discussion forum because of its inherent 
affordances for providing embedded analytics. The Starburst discussion forum (Marbouti, 2012) was 
developed to present discussion threads as a hyperbolic (radial) tree structure, allowing students to see 
the structure of the discussion and the location of their comments within it (see Figure 1). Posts are 
represented as scaled, coloured spheres connected by lines indicating their reply relations. When a post 
is selected, it enlarges to the maximum size and moves to the centre of the diagram while the other 
posts are rearranged around it; sphere size decreases with distance from the currently selected post. For 
each student, new posts are shown in red and previously viewed posts are shown in blue. The initial 
(seed) post always remains yellow. The design rationale and general benefits over traditional linear text- 
based forums have been described previously (Marbouti, 2012) and include more purposeful reading 
and replying behaviours by students, the ability to focus on a part of the discussion in the context of the 
whole, and increased reading of posts in a connected fashion. Easy navigation between connected posts 
also provides context as needed for each comment, thus the reply tool in Starburst omits the commonly 
provided auto-quote function to encourage students who use direct quotes from previous posts to do so 
purposefully. 

In addition to these general benefits for interaction, Starburst also provides an embedded visual analytic 
of listening and speaking behaviours in terms of how the group's discussion is proceeding and how each 
individual is attending and contributing to it. At the group level, the graphical representation of the 
tree's branches allows for easy inspection of the structure of the discussion. Learners can see how many 


1 While the metaphoric notion of a dashboard suggests that one can pay attention to both the activity (driving) 
and instrumentation panel (dashboard) simultaneously, or at least in very quick succession, in practice learning 
analytics dashboards are generally presented as dedicated screens, apart from the main learning activity 
interface. 
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different threads have been created thus far and how deep each is, using this information to inform 
their decisions about where to read and contribute (Marbouti, 2012). They can also examine which 
threads and posts are receiving the most attention (responses) and if any are being neglected. For 
example, in Figure 1 the post labelled "Transfer"...? has not yet been taken up by the group. This kind of 
analytic can be useful in addressing the problem of inadvertent thread abandonment or death (Flewitt, 
2003). 


Menu Reset Refresh 



Welcome ( Logout ) 


Week 12 Discussion 

By , Sep 1, , 9:00 am 

This is the space to discuss the Week 12 reading by Schaffer 8 
Clinton about situated learning and technology design. 

- Reply to: Week 12 Discussion- 

Subject: 


Reply to: Week 12 Discussion 
(or navigate to another post and continue composing your reply) 

I Reply | 


Figure 1. The Starburst discussion tool adapted for analytics 


At the individual level, the red/blue colouring of the posts helps each student easily track which parts of 
the discussion they have attended to already and which parts they have not. For example, in Figure 1 
the student has been heavily attending to one thread (top left), moderately attending to a second (top 
right), and very minimally attending to a third (bottom). In addition, for the current application, we 
made an adaptation from the previous version of the forum to colour the active learner's posts 
differently (light blue). In this way, we provide an analytic to each student of how he or she has 
contributed to the discussion thus far in terms of quantity (high or low volume compared to overall 
quantity of discussion), breadth (distribution throughout the discussion), and intensity (multiple 
contributions to a specific thread). Students can also easily see to which of their posts others have 
replied and to which they have not. In the example shown in Figure 1, the learner has made five posts in 
two of the three active threads. Several posts have stimulated further discussion, in particular the one 
labelled "Synthesizing." Flere the learner can also see his or her round-trip interaction (Anderson, 2008; 
Henri, 1992) with another student responding with "A question." The learner can also see that no one 
has yet addressed the post entitled "A concern." 
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2.3.2 Extracted Analytics 

In contrast to the embedded analytics described above, other useful information about student online 
discussion activity does not easily lend itself to presentation through the graphical interface (e.g., 
temporal distribution of participation). Thus, in our work with extracted analytics, we sought to make 
log-file trace data of speaking and listening activity visible to learners. The metrics selected were 
developed based on our prior research investigating how students attend to the messages of others in 
online discussions described earlier (Wise, Hausknecht, & Zhao, 2014; Wise, Perera et al., 2012; Wise, 
Speer et al., 2013; Wise, Hsiao et al., 2012) and are summarized in Table 1. 


The metrics chosen for the extracted analytics were a subset of indices from the most important 
dimensions/variables identified in the research program that provided a useful signal with a minimum of 
noise (leading to an emphasis on count rather than time-based measures; see Wise, Hausknecht, & 
Zhao, 2014). We also specifically chose metrics thought to be readily understandable by students (with 
brief explanation), and that were complementary to rather than duplicative of the embedded analytics. 
There were, however, two intentional overlaps. First, the metric Number of Posts Made is viewable 
through the embedded analytics; however, the total number of posts made is less salient than their 
distribution, thus providing this sum and the average for the group (not easily determinable from the 
interface) was deemed useful additional information. Second, the metric Percent of Posts Read is similar 
to the embedded red/blue colour tracking of posts viewed in the interface; however, while the interface 
tracks all posts viewed, this metric is only based on posts actually read (not scanned) and thus provides 
complementary (and often quite different) information. 


Table 1. Summary of discussion participation metrics 


Metric 

Definition 

Participation Criteria 

Range of 
Participation 

Span of days a student logged in to the discussion. 

Temporal distribution 

Number of 

Sessions 

Number of times a student logged in to the discussion. 

Temporal distribution 

Percent of Sessions 

with Posts 

Number of sessions in which a student made a post, divided by 
his/her total number of sessions. 

Temporal distribution 

Average Session 
Length 

Total length of time spent in the discussion divided by 
his/her number of sessions. 

Temporal distribution 

Number of Posts 

Made 

Total number of posts a student contributed to the 
discussion. 

Speaking quantity 

Average Post 
Length 

Total number of words posted by a student divided by 
the number of posts he/she made (including manually 
inserted direct quotes). 

Speaking quantity 

Percent of Posts 

Read 

Number of unique posts that a student read divided by 
the total number of posts made by others to the 
discussion. 

Listening breadth 

Number of Reviews 

of Own Posts 

Number of times a student reread posts that he/she 
had made previously. 

Listening reflectivity 

Number of Reviews 

of Others' Posts 

Number of times a student reread others' posts that 
he/she had viewed previously. 

Listening reflectivity 
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Data processing of these analytics is implemented using a toolkit consisting of a combination of mySQL 
queries and Excel VBA macros. Initially, log-file and posts tables are extracted from the discussion forum 
database and merged into a single spreadsheet file. This file lists each action taken by a student in the 
system in a row with the following information: action type (view-post, create-post, edit-post, delete- 
post), a time-date stamp, ID of user performing the action, ID of post being acted on, length of post 
being acted on, and ID of user who created post being acted on. A series of macros then clean the data, 
separate data by user, calculate action duration (subtracting sequential time stamps), divide actions into 
sessions-of-use (based on a 60-minute abandonment threshold, see Wise, Speer et al., 2013), and make 
adjusted estimates for duration of session-ending actions (based on the relevant post's length and the 
average speed of the indicated action by that user). View actions made on a user's own posts are re¬ 
coded as self-review actions and all view and review actions are sub-categorized as reads or scans based 
on a maximum reading speed of eight words per second; a conservative estimate based on Hewitt, 
Brett, & Peters (2007). Finally, the nine variables are calculated based on the definitions shown in Table 
1. A summary table of the metrics is then created for each learner (see Figure 2). This form was chosen 
as a straightforward way of presenting the analytics not requiring particular visual literacy skills and thus 
unlikely to cause confusion. It also allows for concise presentation of multiple metrics. In future work it 
may also be valuable to explore other graphical ways to represent the metrics. In addition to each 
student's individual metrics, we also provide class averages as a local reference point. Reference points 
of greater aggregation (e.g., for other sections of the class, historical averages) could also be included as 
desired. 


Metric 

Your Data (Week X) 

Class Average (Week X) 

Range of Participation 

4 days 

5 days 

Number of Sessions 

6 

13 

Average Session Length 

33 min 

48 min 

Percentage of Sessions with Posts 

67% 

49% 

Number of Posts Made 

8 

12 

Average Post Length 

149 words 

125 words 

Percentage of Posts Read 

82% 

87% 

Number of Reviews of Own Posts 

22 

13 

Number of Reviews of Others' Posts 

8 

112 


Figure 2. Sample learner analytics summary 


2.4 Intervention Design 

As discussed above, an effective learning analytics intervention involves more than simply providing 
well-designed metrics to learners; it necessitates setting up a frame for their use as an integral part of 
course activity tied to goals and expectations (Wise, 2014). The design of our analytics intervention was 
based on the principles of Integration, Diversity, Agency, Reflection, and Dialogue (Wise, Zhao, & 
Hausknecht, 2013). These are used to address possible concerns about rigidity of analytics 
interpretation, hegemony of optimizing to only that which can be measured, and impediment of 
learner's development of metacognitive and self-regulative learning skills (Buckingham Shum & 
Ferguson, 2012; Clow, 2012; Duval & Verbert, 2012). 
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Integration of analytics with the expectations of the learning activity and fabric of the course experience 
is critical for the analytics to be used by students in meaningful ways. Our intervention design helps 
students to understand the goals of the learning activity by providing clear guidelines about the online 
discussions, their purpose, and what is expected of students in terms of quantity, quality, and timing of 
posting, as well as broad, deep, integrated, and reflective attention to the posts of others (see Figure 3a). 
Analytics are then introduced in the context of this framework. For the embedded analytics, the 
appropriate sections of the participation guidelines describe how the interface provides information 
about particular elements of participation (see Figure 3a); for the extracted analytics, a separate 
guideline sheet is given with a chart describing each metric and how it relates to the participation 
criteria (see Figure 3b). In this way, metrics are not presented simply as a set of numbers, but ones that 
have clear meaning in the context of the learning activity. 


a) 


Discussion Participation Guidelines 
Attending to Others’ Posts 

Broad Listening-. Try to read as many posts as 
possible to consider everyone’s ideas in the discussion. 
This can help you examine and support your own ideas 
more deeply. However, when time is limited it is better 
to view a portion in depth, then everything superficially. 

*The visual interface shows posts that you have viewed 
in blue and new ones in red to help you track this. 


Learning Analytics Guidelines 


Attending to Others’ Posts 


% of 

The proportion of posts you read (not scanned) at 


Posts 

least once. 


Read 

It is good to read as many posts as possible to 
consider everyone’s ideas in the discussion 
However, when time is limited it is better to view a 
portion in depth, then everything superficially. 






Figure 3. Excerpt from (a) discussion participation guidelines and (b) learning analytics guidelines 


Diversity in analytic metrics and Agency in interpreting their meaning in the context of the learning 
activity are important in shaping analytics as an empowering tool rather than an exacting master. Our 
intervention is designed to include multiple measures (as described above; see Figure 2). Importantly, 
the guidelines present the metrics as a starting point for consideration, not as absolute arbiters of one's 
engagement in the activity. To further support individual student agency in using the analytics, the 
intervention design has students keep an online reflective journal in which they are encouraged to set 
personal goals for participation and to use the analytics to help monitor these goals. This activity also 
supports the principle of Reflection, as students are given their analytic metrics in the journal space 
along with goal-setting and reflective prompts (see Figure 4). Students are consciously given time to 
write in their reflective journals and the instructor has access to and the ability to comment on students' 
reflections, supporting the principle of Dialogue around the interpretation of the metrics in the context 
of each student's current goals. 


Week 7 

Below you are provided with some detailed information about your discussion participation for the past week generated from the 
system clickstream data. Please refer to the analytics guideline sheet to aid your interpretation and remember that more is not 
always better. Note anything interesting you see in the data in the observations box and then use this information to help you 
answer the reflection questions. 

1. What do you think went well in the group’s discussion this week? What could be improved? 

2. What do you think went well in your individual discussion participation in this week’s discussion? What could be improved? 

3. How does your participation in this week’s discussion compare to previous weeks? 

4. How well did you do in meeting the goal(s) you set for your participation in this week? How do you know? 

5. Please set at least one goal for your participation in the coming week and write it below. 


Figure 4. Sample reflective journal question prompt 
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3. IMPLEMENTATION AND METHODS 


3.1 Research Questions and Design 

The initial pilot of this learning analytics application was conducted as a semi-automated 
implementation with a small group of learners in an authentic class setting. We argue that such 
lightweight testing of the efficacy of theoretically grounded analytics implemented with the minimally 
necessary toolset is a valuable validation step prior to developing a full-blown learning analytics 
platform. In conducting the implementation, we asked three core research questions: 

1. What were students' reactions to the embedded and extracted analytics? 

2. Did the students use the analytics to try to adjust their participation? If so, how? 

3. Did students' participation behaviours change through the term? If so, how? 

A mixed-methods design was used to answer the research questions. Student reactions to and uses of 
the analytics were assessed through analysis of entries in their reflection journals and interviews 
conducted at the end of the term, while changes in behaviour were identified through direct 
examination of the log-file data. Data was examined for individual students as well as across the class to 
identify both unique instances and overall trends. 

3.2 Ethics Procedures 


Study approval was obtained from the office of research ethics at the researchers' university. The 
research protocol included provisions to ensure secure data collection and storage on university servers, 
and participant confidentiality was guaranteed via aggregated data and/or pseudonyms. After obtaining 
instructor consent, all students in the participating class were informed about the purpose of the study, 
provided with an explanation of what their involvement would entail, and invited to give their consent; 
those choosing to participate were able withdraw from the study at any point, although none did. To 
prevent any possible impact on students' educational experience, the course instructor was not present 
during participant recruitment and did not know the identity of students choosing to participate in the 
study until after the course was completed and grades were submitted. 

3.3 Context, Participants, and Learning Activity 

The implementation was conducted in a semester-long blended graduate seminar on educational 
technology consisting of nine first- or second-year doctoral students and one instructor. The seminar 
met once a week in a face-to-face session with a series of ten week-long online discussions interspersed 
between meetings. The first week of discussion was facilitated by the instructor to model good practice 
and to give students a chance to acclimate to the tool; each subsequent discussion week was facilitated 
by one of the course's nine students. 

At the start of the class, students were introduced to the Starburst discussion forum and engaged in a 
conversation about the goal and purpose of online discussions in the course. They were also provided 
with discussion participation guidelines and an explanation of how the embedded analytics could be 
used to help support these. Finally, an online wiki-based reflective journal was created for each student 
as a space for goal setting and reflection on his or her discussion participation. At the beginning of each 
four-hour class session, students were given 10 to 15 minutes to write in their reflective journals based 
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on the prompts described earlier. Between classes, the instructor was invited to read students' 
comments and respond as needed. For the first half of the term (five week-long discussions), the 
reflections were based solely on students' perceptions of the discussions and the embedded analytics. 
This was done to separate out the effects of the embedded analytics and provide a baseline for 
comparison once the extracted analytics were introduced. In the second half of the term (five more 
week-long discussions), students' extracted analytics were calculated for each discussion week and 
pasted into the reflective journal to serve as part of the prompt for reflection. Both embedded and 
extracted analytics were presented explicitly as objects of reflection to understand and effect change on 
one's discussion participation and not as part of the course evaluation. 


3.4 Data Collection 


3.4.1 Interviews 

Shortly after the end of the term, all nine students and the instructor were each interviewed about their 
experiences using the learning analytics as part of the online discussion activity. The hour-long semi- 
structured interviews consisted of four parts: 1) questions about how the participant understood the 
purpose of online discussion and their participation habits; 2) their reaction to and use of the analytics 
embedded in Starburst; 3) their reaction to and use of the specific metrics provided as extracted 
analytics; 4) their experience using the reflective journal as the framing activity for the analytics. 

3.4.2 System Data 

The log-file data that had been processed into extracted analytics was taken to investigate changes in 
student behaviour in the discussions. The same metrics were also calculated for the discussions in the 
first half of the term (when extracted analytics were not provided to the students). In addition to the 
metrics shown to the learners (see Table 1), two additional metrics used in the research program 
(percentage of others' posts viewed and percentage of scans) were calculated for all ten discussion 
weeks to develop a more thorough understanding of learner behaviour in the discussions. 

3.5 Data Analysis 

3.5.1 Common Themes in Reactions to the Analytics 

All interview recordings were transcribed and imported into Atlas.ti. Any text referring specifically to an 
aspect of the embedded or extracted analytics was marked as relevant text (Auerbach & Silverstein, 
2003). Using an inductive approach (Thomas, 2006), each interview was then initially open-coded for 
repeating ideas by one of the researchers. Files were exchanged and a second researcher reviewed the 
codes, adding additional ones as needed. This ensured that the coding of each interview was informed 
by the full breadth of the corpus and research team. Finally, the researchers worked together with the 
repeating ideas to identify emergent high-level themes and characterize the relationships between them. 

3.5.2 Case Studies of Analytics Use and Changes in Behaviour 

A case study of analytics use, including intended and actual changes in behaviour, was conducted for 
each of the nine students in the class based on their interviews, reflective journals, log-file data, and 
post-contents. Since students were encouraged to set up personal goals and choose different paths to 
reach their goals, it was important to look at how their behaviour changed individually. Importantly, 
given that the learning analytics allowed for multiple possible profiles of productive activity and 
improvement, we would not necessarily expect all student behaviour to change in the same way (Wise, 
Zhao, Flausknecht, 2013). 
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Several analyses were performed as part of the case studies to see how individual students used the 
analytics and how their behaviour changed. First, we looked at the goals students set in their reflection 
journal each week, examining their log-file data to see whether empirically these goals were met. We 
also reviewed the students' own reflections on their progress towards their goals. Second, we charted 
each student's metrics across the first five discussions (only embedded analytics) and last five 
discussions (both embedded and extracted analytics) to identify any changes in behaviour induced by 
the extracted analytics (regardless of relation to specific goals). In doing so, it was important to take 
each student's facilitation week into consideration since this could possibly contribute to any changes 
seen in participation (Wise & Chiu, in press). Finally, to contextualize the cases, we examined students' 
attitudes towards particular analytics from their reflective journals and interviews to find possible 
explanations for their changes in behaviour. After building the case study for each student, we then 
looked across cases for common themes and unique instances of how students used the analytics and 
how their behaviours did or did not change throughout the term. 


4. FINDINGS 


4.1 Reactions to the Analytics 

4.1.1 Usefulness of Embedded Analytics 

Over half the students found the graphical branching representation useful in providing information 
about the structure of the discussion. Specifically they reported that it outlined the flow of the 
conversation, showed the connections between ideas, and provided a big picture of the discussion. 
Using this information, some students reported that they chose to attend to threads that were more 
active because they were curious about what topics people emphasized, while others were attracted to 
isolated threads either out of curiosity as to why they had not received much response or because of the 
reduced effort required. Both approaches indicated that students were using analytics embedded in the 
interface to make decisions about how to interact with the discussion. Two students described using the 
information provided about the structure of the discussion strategically, visiting isolated threads when 
they did not have much time, and tackling the larger threads when they did. Despite these positive 
results, a sizable minority of students found the graphical representation of branching confusing, 
particularly when there were many posts in the forum, they found the interface looked too busy and 
thus did not communicate the structure of the discussion effectively. 

With respect to the colouring of nodes to indicate new, read, and one's own posts, just under half of the 
students said the coloured nodes helped them to be more aware of their participation in the discussion 
and they used this information to direct them to which posts to read first. Specifically several students 
found it useful to identify where they had posted, either to see which parts of the discussion they had 
contributed to already and who had replied to them, or to prioritize reading new posts that replied to 
their own posts. Commenting on the specific colour scheme used for this analytic (red/blue/light blue), 
one student found the red colour of new posts "alarming" and said that it "forced [her] to open each of 
them and try to read," while another suggested that the blue and light blue posts could be more 
strongly differentiated. 

Finally, acknowledging the usefulness of having embedded analytics, several students suggested 
additional metrics that could be added to the tool. For example, one student recommended showing 
people's agreement or disagreement through the interface while another suggested that it might be 
helpful to add other information to show which posts are more important to the discussion. 
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4.1.2 Diverse Reactions to Extracted Analytics 

Students had diverse reactions to the nine extracted analytics introduced in the second half of the 
course. Overall, some students found them useful in providing "hard" numbers with which to assess 
their participation; however, others pointed out that there is much that they do not capture. In general, 
student reflections on the extracted analytics seemed to fall into two classes: 1) validation of things 
they were already aware of and 2) metrics that were surprising. Some of the surprises were positive; for 
example, one person felt she was not contributing enough to the discussions but the metrics showed 
that she was substantially above the class average. Other surprises were negative, however; for 
example, for many people the extracted metric of Percent of Posts Read was substantially lower than 
what they expected based on the embedded red/blue posts-viewed analytic (the difference was due to 
post scanning). Reactions to analytic surprises often had an affective component with students variously 
feeling pleased, upset, wronged, or ashamed by their metrics. 


The two metrics most generally mentioned as useful were Average Post Length and Percent of Posts 
Read. The metric of Average Post Length was almost universally popular among students. Many said 
they preferred reading concise posts because they made the discussion "more of a dialogue and less of 
an essay," and thus this metric helped them assess if they were doing a good job keeping their own 
posts succinct. Interestingly, one student who consistently made very short posts set the contrary goal 
of increasing his/her post length and felt that the Average Post Length metric was useful in monitoring 
progress on this goal. The only student who said that the metric was not useful still mentioned that it 
might be helpful for others to keep track of their post length. 


Many students also described Percent of Posts Read as useful, but it was a more controversial metric. In 
describing its usefulness, students said that the metric encouraged them to read (rather than just 
quickly scan) posts and some felt they became more conscious of the way they read in the forum. The 
majority of students also said that they tried to adjust their reading activity based on the metric. 
However, several students felt that the metric did not always represent their activity accurately; 
specifically, they were surprised that the metric indicated that they read less than they thought they did 
and were defensive about this difference, suggesting that the system failed to "catch" all of their reads 
when they read quickly. Interestingly, the majority of students who felt that their actual reads were not 
measured correctly still said that they found this metric useful. 

Students individually put more or less meaning into the remaining metrics with a small number of 
students mentioning each of the metrics as useful to them. For example, a few students found Range of 
Participation helpful to know, and noted that it was especially interesting to see how much others were 
able to participate. Number of Sessions and Average Session Length were appreciated by one student 
who used them to track time spent in the discussion, saying that a longer period was needed for people 
to "digest and understand threads." Percent of Sessions with Posts was helpful for another student to 
"understand how much time I was ... just reading and how many times I was reading and contributing at 
the same time." Some students who found Number of Posts helpful used it to make sure they were 
making enough posts, while others found that they were posting much more than they realized and 
used it to try to reduce the number of posts they made. Students had particularly mixed responses to 
the Number of Reviews of Others' Posts: two students thought peer review was important because you 
can get something new by reading others' posts again; two students were confused by the concept of 
review; and three students specifically found the concept of review unhelpful because they could 
"mostly be reading someone else kind of once and getting it." Finally, although one person said it was 
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good to see the Number of Reviews of Own Posts and started to do more self-reviews towards the 
middle of the course, most other students did not find this metric particularly helpful. 

4.2 Use of the Analytics 

Participants used the analytics in various ways, often leading to a change in their discussion behaviours. 
The following themes described the main uses of the analytics found. 

4.2.1 Students Used Analytics to Set (Often Recurring) Goals for Participation 

Many students found the metrics useful as an evaluation of their participation and they set goals to 
improve their performance. However, students had different emphases on changing various metrics. As 
one student pointed out, "I think based on the numbers you could... choose the direction how you want 
to go." For example, some students paid attention to adjust their post length: "One of [my goals] being 
to reduce the size, I was going quite a bit over the average in some of them so [I thought] let's reduce 
the number of words per post." Other students focused on improving the percentage of others' posts 
they read by spending more time on each post. 

I found that I wanted the challenge of trying to up the percentage of overall posts 
that I reviewed each week. This also meant slowing down my reading since the data 
would not record a quick read of the information. The overall result was that I 
learned more and was able to get a broader sense of opinion concerning the readings. 

Although collectively students had a variety of goals, each student's particular goals were often 
recurring; that is, students expanded or continued goals from previous weeks rather than creating new 
goals each time. Many explained this by acknowledging that even when they knew what they wanted to 
change, it was hard to follow through; it often took a certain amount of time to adjust their behaviour 
and achieve their goals. This could lead to some frustration, as pointed out by one student: "I thought it 
was useful to reflect, but I found it frustrating to reflect and set goals and then not necessarily be able to 
actually meet them." However, the instructor was more positive about the need for students to work 
over several weeks of discussion to achieve their goals. 

I think we want to be working towards feeling comfortable with each other in how 
we comment in the discussions, but that's something I think I have a greater 
appreciation for—the fact that it has to build over time and so I saw several students 
who said, you know, "I don't feel comfortable participating in a certain way as of yet" 
and then pushing themselves to do that later. 

4.2.2 Change with the Analytics were Not Always Intentional 

While the majority of students set goals and tried to alter their behaviours according to the metrics, 
others showed changes in their discussion participation that did not seem to be intentional. These were 
students who described not finding the metrics particularly valuable or not trusting their accuracy, and 
who seemed unaware (even at the end of the course) that their discussion participation behaviours had 
actually changed. For example, one student focused her discussion efforts on qualitative aspects of the 
post contents not captured by the analytics, but still her Percentage of Posts Read increased and 
remained high after the introduction of the metrics. 
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4.2.3 Using Analytics to Monitor Behaviour 

Students also sometimes used the metrics to monitor, but not necessarily change their behaviour. A 
good example of this was with Average Post Length. Post length was a greatly discussed metric in the 
interviews as no one wanted to read excessively long posts. Many students mentioned focusing on 
keeping their own posts concise from the start of the course (before the extracted analytics were 
introduced) and wanted others to do the same. With the introduction of the metrics, students described 
being able to track their post length more easily. 

I think I was pretty consistent probably all the way through, probably pretty much 
around 100 word post is what I would try and keep to—100-150. 

4.2.4 Using Oneself, Others, and Guidelines as Reference Points for Comparison 

Having a reference point with which to compare one's metrics was important for all the students in 
interpreting their analytics. Three specific reference points were commonly used. First, students 
appreciated having discussion guidelines to set up the scope of their participation and described 
thinking about the guidelines while participating, using them as a touchstone for how they should 
participate in an online discussion. Throughout the course, many students described referring to these 
guidelines as a point of comparison for their participation metrics. 

Second, by far the most common reference point used was other students in the class. All students 
expressed an awareness (both in their journals and in the interviews) of the class average given in the 
metrics, with many individuals appreciating it as a reference point. This comparison allowed for a closer 
look at participation in the context of the class: 

It was good to compare because all of those people I think, except one, are coming 
from the [same] background and so on, and myself coming from the [other] 
background, it shows the difference, it shows the difference. That was good for me to 
analyze. 

For several students who were below the class average, this also usefully drove their participation goals. 
However, comparisons with the class average could also take a negative form depending on the student 
and his/her interpretation of the data. While some students found the average interesting, motivating, 
and helpful, others found it surprising, intimidating, and stressful. As one student expressed it, "Since all 
my numbers are below the average so that makes me feel, 'Oh my gosh, I'm kind of jumping out of this 
class' or something like that. It is kind of a little bit — sometimes depressing." 

Finally, students also used themselves as a reference point, looking at their past goals, metrics, and 
reflections as comparison points for their current metrics. Several students noted in their reflections 
when their metrics had gone up or down from the previous weeks: "Compared to previous week, [my] 
number of reviews of others' posts has been hugely increased (excluding my facilitation week), and I did 
spend more time to read and understand others' posts." Some students also used multiple previous 
journal entries to determine whether change was occurring over time: "Actually reading the [journal 
entries] after two, three weeks to get back and see where you were standing and are you still standing 
on the same foot or you change, yeah, that was maybe interesting." 
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4.2.5 Validating Invisible Activity 

Finally, one of the most valuable uses of using the analytics from the instructor's perspective was that it 
honoured and validated discussion forum activity occurring under the surface. Specifically with respect 
to the metrics capturing listening data, it made her aware of the intense involvement of certain students 
who were very engaged with the discussion but did not always post many comments. It also highlighted 
a lack of listening by some of the vociferous speakers. The instructor thus used the analytics as a starting 
point for dialogue in the reflective journal, recognizing the listening efforts of some, while pointing out 
the need for others to listen more deeply. 

4.3 Changes in Behaviour 

4.3.1 Common Changes in Percentage of Posts Read/Viewed 

The most common change observed in students' discussion behaviour related to the Percentage of Posts 
Read. With the introduction of the extracted analytics in the second half of the term, this metric rose for 
all but two students (who were already at the maximum of 100%). Two specific patterns were seen. First, 
students who had a large gap between the percentage of posts they viewed and the percentage of posts 
they read in the first half of the term (due to scanning behaviour) started reading more of the posts they 
viewed, narrowing the gap in the two metrics (see Figure 5a). Second, students whose percentage of 
posts viewed and read were similar, but low, showed a rise in both metrics (see Figure 5b). For two 
students (not shown), there is a potential confound in interpreting the change seen in percentage of 
posts read because their facilitation week occurred just as the extracted analytics were introduced. Thus, 
their rise in Percent of Posts Read in the second half of the term could be due to the facilitator role, the 
introduction of the analytics, or a synergistic effect of the two. 


Percentage of posts viewed and Percentage of posts viewed and 

read for a student closing the gap read for a student increasing rates 



123456789 10 123456789 10 


Weeks Weeks 

-% viewed -% read * facilitation week 

Figure 5. Percentage of posts viewed and read for students who, after the introduction of the 
extracted analytics, (a) narrowed the gap between posts viewed and read and (b) raised the 

percentage of posts both viewed and read 

4.3.2 Unigue Changes in Other Metrics 

Besides the Percent of Posts Read, some students attempted to change other metrics, often 
accomplishing this to some degree. Flowever, these were mostly smaller one-off changes enacted in 
particular weeks but not sustained over time. For example, one student wanted to extend her Range of 
Participation (days in the discussion) for Week 8, but only increased it from three to four days. Another 
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student wanted to increase his Number of Posts Made in Week 7; he did then contribute 15 posts 
(compared to 11 the week before) but dropped back down the following week. Similar to Percentage of 
Posts Read, other metrics also changed for students even if this was not explicitly set as a goal. For 
example, one student increased her Number of Reviews of Others' Posts over the second half of the 
term even though she did not outline this as one of her goals. 


4.3.3 Change in Strategy 

A few participants found that adjusting their metrics to meet their goals meant adopting new strategies 
for discussion participation. This is noteworthy since, as described earlier, changing strongly ingrained 
behaviours can be hard, thus a higher-level approach, such as a strategy shift, may be a productive tool 
to support change. As one student described: 

This week was interesting for me as I had some online sessions that I spent reading 
threads from start to finish without adding to them. This is not something I had done 
much of before and I found that I could follow the thread much better when I didn't 
stop reading to write something myself. This is reflected in the lack of reviews of my 
own/other's posts — I didn't need to reread postings using this method. The 
downside to this is that by the time I had read the whole thread through, I often 
didn't feel I had much to offer as far as new direction/comments/questions that 
hadn't already been said. So my own posting level may have been restricted. 

5. DISCUSSION 


The most basic yet important finding of this study was that when given access to online discussion 
learning analytics in the context of a framing intervention, students enacted some useful changes to 
their participation based on them. This serves as validation of the potential effectiveness of learning 
analytics for influencing student activity in online discussions. As described above, many students were 
purposeful in their use of the analytics, drawing on them to set, enact, and reflect on goals for their 
discussion forum activity. Being proactively involved and engaged in directing one's own learning is 
thought to support better learning processes and outcomes (Boekaerts, Pintrich & Zeidner, 2005; 
Zimmerman & Schunk, 2001). This is an encouraging finding regarding the possibilities of learning 
analytics to support students in the self-regulated learning processes of planning, monitoring, and 
evaluation (Winne & Hadwin, 2010). 

While many students adjusted their participation according to self-set goals and what they believed was 
beneficial for their learning, sometimes students appeared to change their behaviour involuntarily 
simply due to the presence of the metrics. Such unintentional change may have positive or negative 
consequences depending on the changes enacted. For example, some students who felt that the metrics 
were not particularly helpful still increased the percentage of posts they read in the discussions; 
however, students who find out they are above the class average on certain metrics may involuntarily 
reduce their efforts in these areas, as was noted in at least one case here. This highlights that even when 
metrics are presented in a neutral form and with support for student agency in interpretation, the 
analytics still act as agents in the system with some degree of power (Latour, 2005). Thus while data 
may be provided for the purpose of informing decision-making, we must also always try to anticipate 
how it might inadvertently guide behaviour more directly. 
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This is especially important in the case of embedded analytics used in the moment rather than as part of 
a reflective cycle. In this study, with the help of initial stage-setting guidelines, we saw students use the 
embedded analytics purposefully, making strategic decisions about where to post after seeing the big 
picture of the discussion and where they had participated in it. This may help to avoid some typical 
difficulties found in asynchronous online discussions, such as new post bias (Hewitt, 2003; Wise, 
Marbouti, Hsiao, & Hausknecht, 2012), shallow interaction with others' posts (Thomas, 2002), and not 
being able to see the discussion as a whole due to disorganized threading (Dringus & Ellis, 2005). It is 
certainly possible, however, for embedded analytics to have inadvertent undesirable effects as well. For 
example, if students are simply clicking on all the red dots (unread posts) without any effort to read the 
posts because they find their appearance alarming, it is a distraction from meaningful participation in 
the discussion. This highlights a challenging design problem for listening analytics: how to balance the 
useful aspects of tracking which posts have been read without calling undue attention to new unread 
ones (Marbouti, 2012). 


Turning to the extracted analytics, the most common change that students made after their 
introduction related to the percentage of their peers' posts that they read. It is not certain why this 
particular metric was most influential for students, but it may be because it both related to a clear 
concept they valued (attending to others' ideas) and because it revealed information that surprised 
them (either that they were reading fewer posts than they thought or that their peers were reading 
more). Thus, this analytic may have sparked a need for students to reduce the conflict between the 
value they placed on listening to others and the realization that they were not doing this to an 
appropriate extent. 


For some students, the change in percentage of posts they read was enacted as a "narrowing of the 
gap" between posts viewed and read. In other words, these students made an effort to actually read 
(rather than just scan) the posts they opened. While Percentage of Posts Read originally was 
conceptualized as a measure of listening breadth, in this situation the change in the metric to match the 
percentage of posts viewed indicates an increased depth of listening since students were actually 
reading (not just scanning) a greater percentage of the posts they opened. However, for students 
already reading most of the posts they opened, a rise in the percentage of posts read indicates that they 
were actually attending to a greater number of posts overall — an increase in breadth. This distinction is 
important because a recent study (conducted after this analytics implementation was complete) showed 
that depth, but not breadth, of listening was predictive of richer argumentation in student posts (Wise, 
Hausknecht, & Zhao, 2014). We thus have a situation in which the same analytic resulted in two distinct 
changes in student behaviour with different values for learning. This emphasizes both the importance of 
connecting learning analytics to an empirically validated model and the need for clear usage guidelines 
as part of the intervention design. 

Another listening quality found to be important in Wise, Hausknecht, & Zhao (2014) was reflectivity 
(revisiting previously read posts), which predicted more substantive responses to others' ideas. Within 
the current study, however, students did not particularly value the metric regarding review of others' 
posts. Though the participation guidelines explained the value in revisiting previously read posts to 
provide context for recent posts and to examine how thinking had changed, the majority of students 
either were confused by the concept of reviewing posts or felt that reading a post once was enough. 
This demonstrates that when analytics are based on concepts where the value is not obvious to 
students, they may need particular targeted support in understanding why (and how) to use them. 
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In general, the extent to which particular metrics were valued and used by students appeared to vary 
based on two different elements. First, students tended to put more weight on metrics to which they 
had theoretical buy-in, in other words to which they saw the connection to learning. In this application, 
the major support for developing such connections came from the participation guidelines describing 
the expected behaviours in the discussions, and the learning analytics guidelines describing the rationale 
for each metric. While some of the explanations of the reasons for the metrics were accepted by 
students (perhaps because they were simpler or more intuitive), others were not, as shown by the lack 
of value accorded to listening reflectivity. In future work, we will explore alternative avenues to building 
learner value in specific analytics through both expository and interactive means. Learners may also find 
certain metrics more valuable if they are measured at a smaller unit of analysis than weekly averages 
(Wise, Hausknecht, &Zhao, 2014). 


Another element that appeared to influence student perceptions of the metrics was the transparency of 
measurement. For example, one of the metrics that students found most useful, Average Post Length, 
was based on a relatively straightforward calculation of the number of words in a post. Flowever, 
another metric, Percentage of Posts Read was based on a more complicated algorithm that used a 
reading speed threshold to exclude posts that were only scanned. This analytic was sometimes met with 
skepticism, defensiveness, and denial, especially when the figure provided a negative surprise to 
students. While it is possible that some students read faster than eight words per second (leading to 
underestimation of their percent of posts read), this was chosen as a generous threshold thus it is 
unlikely the case for most. For this reason, emotional components of student responses to the analytics 
are important to consider as a more general phenomenon. This is especially an issue when introducing 
analytics involving more complex and less transparent calculations. For example, measures of speaking 
quality (post contents) were not included in this first iteration of online discussion speaking and listening 
analytics but are certainly critical to the value of online discussions for learning. We would expect 
students to buy-in to these metrics theoretically as important for productive learning discussions, but 
how students react to receiving computer-based (computational linguistic) assessments of their posts' 
qualities (e.g., Rose et al., 2008; Ferguson & Buckingham Shum, 2011) or of the extent to which their 
posts show uptake of the contents of prior posts (Suthers, Dwyer, Medina & Vatrapu, 2010) remains to 
be seen. We suspect that students receiving unexpected negative feedback (those who could benefit 
most from the analytics) may become defensive and question the accuracy of how the analytics are 
produced. This kind of situation may thus require additional intervention and support from an 
instructor. 


A final area for which additional support may be required is the process of change itself. Many students 
noted that changing how they participated in the discussion was difficult, even when they knew what 
they wanted to change and had set goals to this effect. In this case, an extended period for adjusting 
one's behaviours as well as support for this process as part of the intervention design were useful. Other 
learning analytics interventions may also require support for students in making desired changes. One 
value that an instructor could provide in such situations is in suggesting ways the student could shift 
his/her learning strategy (rather than just targeting a specific behaviour) to lead to the desired effects. 
This may be particularly helpful for students who are less skilled in regulating their own learning 
(Nandagopal & Ericsson, 2012). In addition, it is important for the design of the learning analytics 
intervention to acknowledge and support change as a process that occurs over time. As was seen in this 
study, many students valued being able to review their analytics over several successive discussions and 
set recurring goals over multiple weeks. This is important, as learning itself is an evolving developmental 
phenomenon that is temporal in nature (Mercer, 2008). Thus the ability for students easily to review 
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both their current analytics and their historical data as a reference point, as well as to have the freedom 
to work on improvement over multiple analytic cycles seems a valuable attribute for learning analytics 
applications more generally. Similar to EnquiryBlogger (Buckingham Shum & Deakin Crick, 2012), the 
system we designed also allowed students to track their goals and reflections over time, supporting 
awareness of the changes that occurred and allowing for a potentially deeper understanding of the 
process of altering one's activity patterns. 


Besides the metrics for each individual over time, the class average also played an important role in 
influencing student changes in behaviour by providing a contemporaneous comparison point. While the 
class average can be a powerful reference point for low-activity students, due to the power of peer 
influence there is also a danger that its use can lead to a reduction in activity diversity (all members of 
the learning community collapse to a single discussion participation profile). Thus, in future work, it 
would be preferable to provide a range of peer activity as a reference frame to support students in 
determining what changes to make to their discussion participation. Giving a range or gradient of peer 
activity as a reference frame may be best communicated in a graphical form, rather than the numerical 
presentation used here. Another area for future analytics work is using peer activity to provide 
information about post reception; for example, using a measure of how certain posts (or learners) are 
attended to by others (Brooks, Greer, & Gutwin, in press) as a proxy for their influence or importance in 
the discussion. 


6. LIMITATIONS 


The current work has several limitations, primarily related to conducting the pilot in a small context with 
advanced and motivated students. First, the model of reflective journals with one-on-one dialogue 
between the instructor and each student is not sustainable at scale. To replicate this interaction in more 
populous contexts, instructors can consider using peer commenters or consolidating the reflections into 
larger units as a formal assignment. We have implemented the latter approach to reflection (without 
analytics) in a large undergraduate class, suggesting that it would also be viable for this purpose. 
Second, the Average Post Length metric counted all words in a post, incorporating any direct quotations 
taken from prior posts or readings. In the context of a discussion tool without an auto-quote function 
and with advanced students this was not a problem since quotes were generally used sparingly and 
meaningfully. However, with younger, more novice students, overuse of extended direct quotations 
may artificially inflate this metric, requiring refinement of the calculation algorithm. 

Finally, and most importantly, the specific findings in this context may not generalize to the larger 
population of students taking part in online discussions as an educational activity. This is especially true 
for students just starting post-secondary studies, less keen on learning, or studying different kinds of 
subject matter. Further testing with a diverse range of students, subject areas, and instructional levels 
will reveal the extent to which the specific results found here are robust for the larger population of 
online and blended learners (or subsets thereof), and across different learning contexts. As well, we 
expect other patterns to emerge when more student data is available for analysis. In contrast, some of 
the conceptual issues we uncovered relate to putting embedded and extracted online discussion 
analytics into practice more generally. Thus, phenomena such as unintentional discussion behaviour 
change and the presence of affective reactions to the metrics may provide conceptual generalizability 
useful to future online discussion learning analytics efforts. 
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7. CONCLUSION 

This paper has described one effort to develop and test a learning analytics application based on an 
existing educational research program. We presented a theoretical framework and empirical findings 
about students' speaking and listening in online discussions, used this to develop analytics both 
embedded in and extracted from the learning environment. In addition, we designed an intervention to 
frame analytics use as an integral part of the learning activity. Together, the careful design of embedded 
analytics, extracted metrics, and their framing for use contribute to a system that employs learning 
analytics as a guide for sense-making of discussion participation with the goal of empowering students 
to take responsibility for regulating their own learning processes. 

While the results of this study are promising for the use of learning analytics to support student 
participation in asynchronous online discussions, we emphasize that the sample was small and not 
representative of the general population of students taking part in online discussions. Thus, the extent 
to which specific findings (e.g., the usefulness of the embedded analytics, the dominance of changes in 
the percentage of posts read, and the setting of recurring goals) may generalize remains for future work. 
We suggest, however, that the higher-level issues related to putting analytics into practice revealed by 
this study have useful conceptual generalizability. Thus we present the following five concerns to be 
considered in future work regarding online discussion learning analytics targeting students. First, with 
the introduction of the analytics, students may enact not only purposeful, but also unintentional 
changes in their activity. Second, the same analytic can lead to multiple distinct changes in student 
behaviour. Third, for students to use particular analytics they need to value them; this may be affected 
both by their theoretical buy-in to the measure and the transparency/perception of the accuracy of its 
calculation. Fourth, the affective component of students' reactions to the analytics needs to be taken 
into account. Finally, while current analytics systems work to help us understand diagnostically what 
needs to change, students may also require additional support in learning how to enact that change. 
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